CTG-Net: Cross-task guided network for breast ultrasound diagnosis

Deep learning techniques have achieved remarkable success in lesion segmentation and classification between benign and malignant tumors in breast ultrasound images. However, existing studies are predominantly focused on devising efficient neural network-based learning structures to tackle specific tasks individually. By contrast, in clinical practice, sonographers perform segmentation and classification as a whole; they investigate the border contours of the tissue while detecting abnormal masses and performing diagnostic analysis. Performing multiple cognitive tasks simultaneously in this manner facilitates exploitation of the commonalities and differences between tasks. Inspired by this unified recognition process, this study proposes a novel learning scheme, called the cross-task guided network (CTG-Net), for efficient ultrasound breast image understanding. CTG-Net integrates the two most significant tasks in computerized breast lesion pattern investigation: lesion segmentation and tumor classification. Further, it enables the learning of efficient feature representations across tasks from ultrasound images and the task-specific discriminative features that can greatly facilitate lesion detection. This is achieved using task-specific attention models to share the prediction results between tasks. Then, following the guidance of task-specific attention soft masks, the joint feature responses are efficiently calibrated through iterative model training. Finally, a simple feature fusion scheme is used to aggregate the attention-guided features for efficient ultrasound pattern analysis. We performed extensive experimental comparisons on multiple ultrasound datasets. Compared to state-of-the-art multi-task learning approaches, the proposed approach can improve the Dice’s coefficient, true-positive rate of segmentation, AUC, and sensitivity of classification by 11%, 17%, 2%, and 6%, respectively. The results demonstrate that the proposed cross-task guided feature learning framework can effectively fuse the complementary information of ultrasound image segmentation and classification tasks to achieve accurate tumor localization. Thus, it can aid sonographers to detect and diagnose breast cancer.

include information on whether the IRB approved this study, and the name of the IRB. Please also include information on how and when participants provided consent for their data to be used in research." Authors' Reply: We thank you for pointing out this significant issue regarding the dataset preparation. We agree that it is an important issue, and we apologize that we have not clearly presented the related information. We have updated the collection protocols: (1) selection criteria and (2) exclusion criteria to clarify the patient data in the Dataset subsection of the manuscript (pages 6-7, lines 233-249). Please see the following text. {The THH dataset were obtained by breast surgeons with over 17 years of experience using ultrasound equipment from patients who underwent breast ultrasound examinations at Takamatsu Heiwa Hospital between May 2012 and January 2017 and met the following criteria.
• Selection criteria: (1) Those with findings, such as masses and nonmassive lesions in the mammary glands, or without apparent findings; and (2) those who do not refuse to participate in this study.
• Exclusion criteria: Exclusion criteria: (1) patients who have undergone mastectomy; (2) those with substantially thicker or thinner mammary glands or mammary glands with severe mastopathy; (3) those with inadequate samples; and (4) those who are deemed inappropriate as research participants by the principal investigator. This procedure was approved by the Ethics Committee of Takamatsu Heiwa Hospital on January 18, 2018, and by the Ethics Committee of the National Institute of Advanced Industrial Science and Technology on March 9, 2018. The UDIAT and BUSI datasets were provided by [46] and [47], respectively, and were used under institutional or patient approval. Detailed public datasets access information are provided by the Supporting Information S2_data and S3_data.} 4. Please note that PLOS ONE has specific guidelines on code sharing for submissions in which author-generated code underpins the findings in the manuscript. In these cases, all author-generated code must be made available without restrictions upon publication of the work. Please review our guidelines at https://journals.plos.org/plosone/s/materials-and-software-sharing#loc-sharing-code and ensure that your code is shared in a way that follows best practice and facilitates reproducibility and reuse. Authors' Reply: Thank you for the constructive suggestion regarding code sharing. We agree that reproducibility and replicability are of core significance for scientific publications and source code plays a central role to address ambiguity in the research results. Therefore, we prepared source code with full annotations which are consistent to the flowchart and algorithm explanations presented in the manuscript. For details, please see the zip file in the Supporting information subsection (page 18, lines 517-520).

5.
Please review your reference list to ensure that it is complete and correct. If you have cited papers that have been retracted, please include the rationale for doing so in the manuscript text, or remove these references and replace them with relevant current references. Any changes to the reference list should be mentioned in the rebuttal letter that accompanies your revised manuscript. If you need to cite a retracted article, indicate the article's retracted status in the References list and also include a citation and full reference for the retraction notice.

Authors' Reply:
Thank you for the comments to the reference section. We have carefully checked the reference list and updated it to ensure its completeness and correctness. We have highlighted these additions in the References section of the revised manuscript (pages 18-23).
The following are the additions to the literature: {References 14, 15, 18, 19, 21, 27-33, and 36-40} Dear reviewer #1, Thank you for your valuable advice. After carefully studying your comment, we have finished the revised manuscript and labeled 'Revised Manuscript with Track Changes'. Below is a description of the point-to-point changes we have made based on your comments. Comment 1. Add in the last lines of Abstract in what %age and in what parameters, the proposed methodology is better as compared to existing techniques and what is the overall analysis of the proposed methodology. Authors' Reply: According to your comments regarding quantitative evaluation, we have revised the content of {We performed extensive experimental comparison on both private and public ultrasound datasets and the results validated that the proposed approach achieved significant improvements compared with the state-of-the-art methods. The results demonstrate the effectiveness of the proposed cross-task guided feature learning framework for ultrasound image recognition, and it can be one potential solution for clinical application in aiding sonographers to detect and diagnose breast cancer.} in the previous manuscript to {We performed extensive experimental comparisons on multiple ultrasound datasets. Compared to state-of-the-art multi-task learning approaches, the proposed approach can improve the Dice's coefficient, true positive rate of segmentation, AUC, and sensitivity of classification by 11%, 17%, 2%, and 6%, respectively. The results demonstrate that the proposed cross-task guided feature learning framework can effectively fuse the complementary information of ultrasound image segmentation and classification tasks to achieve accurate tumor localization. Thus, it can aid sonographers to detect and diagnose breast cancer.} (Abstract section, page 1).

Comment 2.
Under contributions, add one-two points with regard to experimentation. Add Organization of the paper at end of introduction. Authors' Reply: For the top level design of our experimental validation, we mainly addressed two points: 1. We do fair comparison with other latest MTL methods (page 3, lines 78-80). 2. The tests were performed upon multiple datasets and thus we avoid data-induced bias in performance comparison (page 3, lines 81-83). Furthermore, we presented all the details of experiments in revised manuscript, including dataset (pages 5-7, lines 189-249), parameter settings (page 11, lines 353-365), evaluation protocol (page 11, lines 366-372) and ablation study scheme (pages 16-17, lines 441-494). From fundamental design to implementation details, we organized those contents in a logical manner to ease understanding. Specifically, we have added one point with regard to experimentation under contributions (page 3, lines 81-83). The additions in the revised manuscript are as follows: {

•
The proposed approach achieves excellent performance on several private and public datasets with visual differences proving that the proposed approach has good generalization performance and can minimize bias caused by the dataset.} In addition, we have also added the organization of this paper at the end of the introduction (page 3, lines 84-89), please see as follows： {The rest of the paper is organized as follows. Section 2 introduces the related work to the proposed method, Section 3 describes the datasets adopted and explains the overall structure of the proposed method, component units, and loss function. Section 4 presents the experimental setup, evaluation metrics, and experimental results. Section 5 discusses the ablation experiments and failure cases. Finally, this study is concluded in Section 6.} Comment 3. Under literature review, it is suggested to add min 15-25 papers which are latest and taken as base for the proposal of methodology, and every paper should be elaborated with what is proposed, what is the novelty and what experimental results are there. At the end of Literature review, highlight in 9-15 lines what overall technical gaps are there in the paper, that led to the design of proposed methodology. Authors' Reply: According to your comments regarding the literature review, we have made substantial revision to the current reference list, and the current version covers representative methods for BUS image analysis, some of which are used as baselines for experimental comparisons. Concretely, we have added more recent literature in accordance with your comment. These are the literature recommended in comment 9 and the latest literature about multi-task learning. The following are the additions to the literature: References 14,15,18,19,21,27 [36] used the VNet architecture to develop a CAD system that can jointly perform 3D automatic breast ultrasound (ABUS) image classification and segmentation CAD system. They exploited the extracted multi-scale features to improve the image classification task and achieve better results than a single task through an iterative feature refinement strategy. Zhang et al. [37] proposed BI-RADS-Net for explainable BUS CAD based on multi-task learning. The model outputs the probability of class and malignancy of a tumor by performing multiple classification and regression tasks. Cao et al. [38] proposed a multi-task learning method based on label distribution correction for overcoming the problem of insufficient labeled training data. They performed breast tumor classification task jointly using two labels from different domains of expertise and demonstrated the effectiveness of the method on the collected dataset.} In addition, based on the literature review, we have provided the following analysis to illustrate the advantages of the proposed method. {The above survey reveals that the MTL approach could be a promising approach, however, there was little in-depth investigation along this research direction. In contrast to previous studies, our contribution is three-fold, which are as follows. First, it is acknowledged that finding suitable auxiliary tasks plays the most important role for MTL. The tasks should have some level of correlation, otherwise, training on irrelevant tasks can result in negative transfer and deteriorate the performance. To the best of our knowledge, this is the first study to formulate lesion classification and its region segmentation as a multi-task learning problem for BUS image analysis. The two tasks are highly correlated and thus appropriate to be investigated through multi-task learning. Second, to achieve superior performance in lesion classification and segmentation, we adopted the attention mechanism in the proposed neural network design, which enables the network to focus on a few particular aspects that are related to suspicious lesion areas and ignore the rest. In other words, it is an integral building block to generating pixel-wise labels for the lesion region. Third, MTL has been commonly formulated as a minimization of a linear combination of individual tasks' loss functions. The task-specific weights are critical parameters to tune through the learning process. We adopted a self-adjusted scheme to estimate the task-specific weights through optimization, which is more efficient and robust compared to conventional methods such as grid search through cross-validation.} (page 5, lines 157-175) Comment 4. Under methods, add the methods, which are used to design the proposed methodology. Authors' Reply: We have checked the manuscript carefully and added further details regarding the design of the proposed method (page 7, lines 258-265). The added materials illustrate how our proposed approach ensures that segmentation and classification can be effectively facilitated between them. {Classification and segmentation can achieve mutual complementarity based on two critical evidences: (1)  Comment 5. Add proper system model of the proposed methodology. Add algorithm and flowchart of the proposed methodology Authors' Reply: According to your suggestions regarding the proposed methodology, we revised our paper as follows. In the Methodology section (from page 7) of the manuscript, we have described the system model of the proposed method in detail. Moreover, the flowchart of the proposed methodology has been presented in Fig. 5. We have added the algorithmic pseudo-code (page 12) for model optimization and modified the manuscript (page 11, lines 364-365) below to further illustrate the approach in this study. {Algorithm 1 provides the algorithm details to clearly show the optimization process of our proposed method.} We believe that Algorithm 1 clarifies doubts regarding the algorithm implementation. The following figure shows Algorithm 1. Comment 6. Add Analysis section to the paper. Authors' Reply: We apologize for the unclear expression due to writing problems. To clearly explain the analysis section in our manuscript, we have revised the text related to the analysis in the revised manuscript. The following are the contents related to the analysis. 1) Comparative experimental analysis to previous methods. We conducted extensive comparative experiments with state-of-the-art segmentation, classification, and multi-task learning methods to analyze the superiority of the proposed method. We have modified the grammatical writing of this part to make it easier to understand. Detailed experimental results and specific analysis are presented in the Comparisons with state-of-the-art methods subsection (pages 13-15, lines 375-439). 2) Error analysis and discussion for future improvement. To analyze the limitations of the proposed method, we have performed a detailed analysis based on some cases of recognition failure (Limitations subsection, page 17, lines 481-489), and added the limitations of the proposed method (Limitations subsection, page 17, lines 490-493). In addition, we also added directions for future work based on potential solutions (at the end of the Conclusion section, page 18, lines 507-511). 3) Overall performance. We performed an exhaustive ablation experiment to analyze the effectiveness of our proposed overall structure and individual modules in Discussions section. For example, as discussed in the Ablation study subsection (page 16, lines 444-449), we compared and analyzed the overall performance of the network without using any module to the proposed method.
Comment 7. Add some case study based discussion to the paper. Authors' Reply: We have accordingly presented the discussed cases from several aspects and modified the grammatical writing to further facilitate understanding. 1) We have revised the Comparisons with state-of-the-art methods section of our manuscript, and selected cases for discussion in the private dataset (pages 13-14, lines 385-388, 395-399 illustrated in Fig. 9) and public datasets (page 14, lines 412-420 illustrated in Fig. 10), respectively. These case studies show the advantages of the proposed method over other stateof-the-art methods. 2) We have discussed in detail the four failed cases in Fig. 11  There is no definite conclusion on which optimizer should be used in deep learning model training. Adam is used in this study for the following two reasons: (1) Adam has the advantages of adaptive learning rate, and simple and efficient implementation, and it is one of the most commonly used optimizers. Furthermore, this study [1] demonstrated that ADAM can be generally better than SGD.
(2) In previous experiments, we used Adam, SGD, and Adagrad optimizers for comparison, and the final results show that Adam performs better for this task.
[1] Choi, Dami, et al. "On empirical comparisons of optimizers for deep learning." arXiv preprint arXiv:1910.05446 (2019). We mentioned content related to this comment in the Experiment setup subsection (page 11, lines 357-361). The additions in the revised manuscript are as follows: {Although better optimization methods have been proposed, the Adam optimizer has a simple mechanism and is often used as a standard optimization method by many methods. Therefore, this study uses the Adam optimizer to assess the intrinsic superiority of the proposed approach by checking whether the standard optimization method can also obtain good performance.} Comment 2. References should be updated with recent works related to the proposed study Authors' Reply: We have updated more recent literature based on your comments. The following is the list of the added literature: {References 14, 15, 18, 19, 21, 27-33, and 36-40} The following is a detailed description: